MapReduce-Style Computation in Distributed Virtual Memory

Many cloud computing technologies, such as MapReduce, use file systems as the system-wide substrate for data handling. A distributed file system provides a global name space and stores data persistently, but it also introduces significant overhead. Several recent systems use DRAM to store data and tremendously improve the performance of cloud computing systems. However, both our own experience and related work indicate that a simple substitution of distributed DRAM for the file system does not provide a solid and viable foundation for data storage and processing in the datacenter environment, and the capacity of such systems is limited by the amount of physical memory in the cluster. To overcome the challenge, we construct a distributed virtual memory to unify the physical memories and disk resources on many compute nodes to form a system-wide data substrate. The new substrate provides a general memorybased abstraction, takes advantage of DRAM in the system to accelerate computation, and, transparent to programmers, scales the system to handle large datasets by swapping data to disks and remote servers. Above the distributed virtual memory, we develop vMR, a new design of the MapReduce framework with enhanced functionality enabled by the memory-based abstraction. The evaluation results show that vMR is faster than Hadoop on all the 4 workloads, and delivers 6-11x speedups on the adjacency list workload. For iterative k-means clustering, vMR can run more than 7 times faster than Hadoop and scale linearly to 160 compute nodes on the TH-1/GZ supercomputer.